feat: Durable Web Search golem:web-search API Across Different Providers in Rust #66

Rutik7066 · 2025-07-08T12:26:59Z

Added Tavily, Serper, Brave & Google custom search.
Demo link : https://drive.google.com/file/d/16RxMlgtNzEp9Z1vYN-JZskMXBHARhuAQ/view?usp=sharing
Note: Microsoft Bing is not added. Microsoft will discontinue the bing api support after 11th August 2025
Ref : https://www.microsoft.com/en-us/bing/apis/bing-web-search-api

claim #34
/fixes #34

Rutik7066 · 2025-07-08T12:29:32Z

My last PR was closed at this link #46. I accidentally deleted the branch from GitHub, which caused the PR to close automatically. Fortunately, I had the branch saved locally, so I opened this new PR.

mschuwalow · 2025-07-17T18:18:34Z

web-search/brave/src/lib.rs

+
+        let page_size = self.params.max_results.unwrap_or(10);
+        let current_offset = *self.current_offset.borrow();
+        let new_offset = current_offset + page_size;


Looking at the brave docs the offset should be incremented by 1 to get the next page of resuts, not page_size:
https://api-dashboard.search.brave.com/app/documentation/web-search/query#WebSearchAPIQueryParameters

mschuwalow · 2025-07-17T18:26:14Z

web-search/google/src/lib.rs

+                "No more results available".to_string(),
+            ));
+        }
+        *self.current_start_index.borrow_mut() = *self.current_start_index.borrow_mut() + 1_u32;


This should be incremented by num, not by 1: https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list#response

mschuwalow · 2025-07-17T18:38:53Z

Requested access to the demo video, please grant access

.github/workflows/ci.yaml

Rutik7066 · 2025-07-17T18:57:20Z

@mschuwalow, granted access for the video. Please check, and I will update changes asap

Rutik7066 · 2025-07-20T18:03:01Z

Ohh, I understood now. That is why when we transition to live, it starts from the beginning.

Rutik7066 · 2025-07-20T18:25:24Z

@mschuwalow Please check i updated the changes

.github/workflows/ci.yaml

Rutik7066 · 2025-07-21T10:48:51Z

@mschuwalow Are we ready to merge?

mschuwalow · 2025-07-21T10:52:34Z

web-search/brave/src/lib.rs

+        println!("[DURABILITY] unwrapped_search_session: Creating new BraveSearchSession");
+        LOGGING_STATE.with_borrow_mut(|state| state.init());
+
+        with_config_key(&[Self::API_KEY_ENV_VAR], Err, |keys| {


Minor: please move reading the env var into the next_page call.

Otherwise a restored session might use a different api key then a live session (as they end up reading the environment variables at different times). This might make sense, but for now we want to avoid divergence between replay as live as much as possible

We will remove the env reading and eliminate that function. I'm going to take out unwrapped_search_session and move the env reading to next_page, which will introduce some overhead due to creating a new client and reading env each time. However, it's a necessary adjustment.

web-search/web-search/src/durability.rs

mschuwalow · 2025-07-21T11:02:44Z

web-search/brave/src/lib.rs

+        let (results, metadata) =
+            convert_response_to_results(response, &self.params, Some(current_offset));
+
+        *self.last_metadata.borrow_mut() = metadata.clone();
+
+        if results.is_empty() {
+            *self.has_more_results.borrow_mut() = false;
+            return Err(SearchError::BackendError("No more results".to_string()));
+        }
+
+        if let Some(metadata) = &metadata {
+            let api_has_more = metadata.next_page_token.is_some();
+            let within_limits = new_offset < 9;
+            *self.has_more_results.borrow_mut() = api_has_more && within_limits;
+        } else {
+            *self.has_more_results.borrow_mut() = false;
+        }
+        *self.current_page.borrow_mut() += 1; 
+        results
+            .into_iter()
+            .next()
+            .ok_or_else(|| SearchError::BackendError("No results returned".to_string()))


The logic that you execute after the request is sent to update the internal state will need to be reproduced in session_for_page, as otherwise the internal state of the session will be different for live and replay (which will lead to different results).

I think your life will be easier if you move all checks / calculations into the beginning of next_page if possible. That way both replay and live should behave consistently. If this is not possible, you might need to persist additional state as part of replay so you can restore the internal session state.

Hey, I have a solution for this. Since we are tracing the page_count in durable layer, then we can eliminate calculations from the next_page and create a separate method in extendedtrait to update pagination state with the tracked page count from durable. In the case of replay, we will increase page_count by one so that it will start exactly where the crash happened

Rutik7066 · 2025-07-21T11:12:19Z

Okay I will update asap

…hod across multiple web search components to streamline session management.

Rutik7066 · 2025-07-21T12:27:17Z

@mschuwalow updated the changes please take a look.

Rutik7066 · 2025-07-21T13:04:10Z

will complete exa asap

Rutik7066 · 2025-07-21T14:45:07Z

@mschuwalow Please check now i have addressed all the comments.

mschuwalow · 2025-07-21T14:54:48Z

web-search/web-search/src/durability.rs

+                let state = self.state.borrow();
+                let result = match &*state {
+                    Some(DurableSearchSessionState::Live { session, .. }) => session.get_metadata(),
+                    _ => None,


This case needs to be handled (we just finished replay and the user asks us for the current metadata).

mschuwalow · 2025-07-21T14:55:33Z

web-search/web-search/src/durability.rs

+                            with_persistence_level(PersistenceLevel::PersistNothing, || {
+                                session.next_page()
+                            });
+                        let persisted_result = durability.persist_infallible(NoInput, result.clone());


You should you use perist here instead of persist_infallible

@mschuwalow I am trying to understand the difference between persist_infallible and persist, but I cannot find any documentation on that. Could you please explain?

Rutik7066 · 2025-07-21T20:00:16Z

For now, I am reverting EXA since it does not support pagination.

Rutik7066 · 2025-07-21T21:21:48Z

@mschuwalow Updated the changes. Please take a look.

mschuwalow · 2025-07-22T09:21:01Z

I decided to go with #60, as it's closer to being merged.

Thank you for your work on this, I can tell you put a lot of work into it.

Rutik7066 · 2025-07-22T09:49:22Z

@mschuwalow, what are you still asking for changes on that PR, and closed mine. Did you look at the code at least after changes?

Rutik7066 · 2025-07-22T09:52:46Z

@mschuwalow Last time you did the same #2. My implementation was good from a maintenance perspective as well. But you still chose that PR.
you still can check and compare, which one is better and close.
cc @jdegoes @vigoo

Rutik7066 added 17 commits June 24, 2025 20:27

web-search crate

53c9bb2

google

453ad6a

added test

54ddb9f

test build error fix

5f10ce5

wip

40c7d13

final

d6f56a0

ci fix

d385745

Merge remote-tracking branch 'origin/main' into web-search

0b32512

rebase wip

cc2851e

fix

3493156

ci fix

c08754d

revert

1117ae9

fix

e9c3021

contry and language code mapping

24cb705

lint

6d9ca57

Trigger PR reopen

230b5f3

Trigger PR reopen

fee5b3e

mschuwalow requested changes Jul 17, 2025

View reviewed changes

mschuwalow reviewed Jul 17, 2025

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

Rutik7066 added 6 commits July 17, 2025 19:43

brave fix

99d37a6

google fix

8140ea5

ci ollama fix

192bf4e

Merge remote-tracking branch 'origin/main' into web-search

c71e3df

clippy

24e6db4

revert ci ollama fix

88af836

Rutik7066 requested a review from mschuwalow July 17, 2025 20:27

unit test fix

887b442

update count_page and finished

baedbf1

mschuwalow reviewed Jul 21, 2025

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

Update .github/workflows/ci.yaml

df00b4c

mschuwalow requested changes Jul 21, 2025

View reviewed changes

Rutik7066 added 3 commits July 21, 2025 11:26

get_meatdata durable

b9d0e5d

Refactor ExtendedGuest trait by removing unwrapped_search_session met…

53f443a

…hod across multiple web search components to streamline session management.

refactor

2c06807

Rutik7066 added 2 commits July 21, 2025 12:56

new provider exa wip

587b9c7

exa wip

cb7aea4

Merge remote-tracking branch 'origin/main' into web-search

3d52151

mschuwalow requested changes Jul 21, 2025

View reviewed changes

Rutik7066 added 3 commits July 21, 2025 15:32

replay case get metadata

7e12c4a

updated next_page to use failable durabilty api

08cc7fb

clippy

23ba0b3

Rutik7066 force-pushed the web-search branch from 5e50da0 to 23ba0b3 Compare July 21, 2025 18:19

removed exa

456c7d6

mschuwalow closed this Jul 22, 2025

Rutik7066 deleted the web-search branch July 23, 2025 14:32

feat: Durable Web Search golem:web-search API Across Different Providers in Rust #66

feat: Durable Web Search golem:web-search API Across Different Providers in Rust #66

Uh oh!

Conversation

Rutik7066 commented Jul 8, 2025

Uh oh!

Rutik7066 commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschuwalow commented Jul 17, 2025

Uh oh!

Uh oh!

Rutik7066 commented Jul 17, 2025

Uh oh!

Rutik7066 commented Jul 20, 2025

Uh oh!

Rutik7066 commented Jul 20, 2025

Uh oh!

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

Rutik7066 commented Jul 21, 2025

Uh oh!

mschuwalow commented Jul 22, 2025

Uh oh!

Rutik7066 commented Jul 22, 2025

Uh oh!

Rutik7066 commented Jul 22, 2025

Uh oh!

Uh oh!